Array Cameras Incorporating Independently Aligned Lens Stacks

ABSTRACT

Array cameras, and array camera modules incorporating independently aligned lens stacks are disclosed. Processes for manufacturing array camera modules including independently aligned lens stacks can include: forming at least one hole in at least one carrier; mounting the at least one carrier relative to at least one sensor so that light passing through the at least one hole in the at least one carrier is incident on a plurality of focal planes formed by arrays of pixels on the at least one sensor; and independently mounting a plurality of lens barrels to the at least one carrier, so that a lens stack in each lens barrel directs light through the at least one hole in the at least one carrier and focuses the light onto one of the plurality of focal planes.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. patent applicationSer. No. 14/536,537 entitled “Methods of Manufacturing Array CameraModules Incorporating Independently Aligned Lens Stacks” to Rodda etal., filed Nov. 7, 2014, which application claims priority under 35U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No.61/901,378 entitled “Non-Monolithic 3−3 Array Module with DiscreteSensors and Discrete Lenses” to Rodda et al., filed Nov. 7, 2013 andU.S. Provisional Patent Application Ser. No. 61/904,947 entitled “ArrayCamera Modules and Methods of Manufacturing Array Camera ModulesIncorporating Independently Aligned Lens Stacks” to Rodda et al., filedNov. 15, 2013. The disclosures of U.S. Provisional Patent ApplicationSer. Nos. 61/901,378 and 61/904,947 are hereby incorporated by referencein their entirety.

FIELD OF THE INVENTION

The present application relates generally to array cameras and morespecifically to array cameras incorporating independently aligned lensstacks and physically discrete sensors forming an array, a single focalplane sensor utilizing a virtual array, or a monolithic sensor havingmultiple physical focal planes.

BACKGROUND

Imaging devices, such as cameras, can be used to capture images ofportions of the electromagnetic spectrum, such as the visible lightspectrum, incident upon an image sensor. For ease of discussion, theterm light is generically used to cover radiation across the entireelectromagnetic spectrum. In a typical imaging device, light entersthrough an opening (aperture) at one end of the imaging device and isdirected to an image sensor by one or more optical elements such aslenses. The image sensor includes pixels or sensor elements thatgenerate signals upon receiving light via the optical element. Commonlyused image sensors include charge-coupled device (CCDs) sensors andcomplementary metal-oxide semiconductor (CMOS) sensors.

Image sensors are devices capable of converting an optical image into adigital signal. Image sensors utilized in digital cameras are made up ofan array of pixels. Each pixel in an image sensor is capable ofcapturing light and converting the captured light into electricalsignals. In order to separate the colors of light and capture a colorimage, a Bayer filter is often placed over the image sensor, filteringthe incoming light into its red, blue, and green (RGB) components whichare then captured by the image sensor. The RGB signal captured by theimage sensor plus Bayer filter can then be processed and a color imagecan be created.

Generally, image capture utilizes a single image sensor, to captureindividual images, one at a time. A digital camera typically combinesboth an image sensor and processing capabilities. When the digitalcamera takes a photograph, the data captured by the image sensor isprovided to the processor by the image sensor. Processors are able tocontrol aspects of a captured image by changing image capture parametersof the sensor elements or groups of sensor elements used to capture theimage.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of the inventioninclude processes for constructing array camera modules, array cameramodules, and array cameras that include multiple lens stacks separatelymounted to a carrier.

One embodiment includes: forming at least one hole in at least onecarrier; mounting the at least one carrier relative to at least onesensor so that light passing through the at least one hole in the atleast one carrier is incident on a plurality of focal planes formed byarrays of pixels on the at least one sensor; independently mounting aplurality of lens barrels to the at least one carrier, so that a lensstack in each lens barrel directs light through the at least one hole inthe at least one carrier and focuses the light onto one of the pluralityof focal planes; and mounting a module cap over the lens barrels, wherethe module cap includes at least one opening that admits light into thelens stacks contained within the plurality of lens barrels.

In a further embodiment, forming at least one hole in at least onecarrier includes forming at least one hole in a single carrier.

In another embodiment, mounting the single carrier relative to at leastone sensor includes mounting the single carrier relative to a pluralityof sensors.

In a still further embodiment, each of the plurality of sensors ismounted to a first side of the single carrier; each of the plurality oflens barrels is mounted to a second opposite side of the single carrier;and the plurality of sensors comprises a separate sensor for each of theplurality of lens barrels.

In still another embodiment, the at least one hole in the single carrierare spaced to enable an active alignment tool to grip the lens barrelduring the active alignment process.

In a yet further embodiment, flip chip mounting is utilized to mount theplurality of sensors to the single carrier.

In yet another embodiment, the plurality of sensors is mounted to asubstrate and mounting the single carrier relative to the plurality ofsensors comprises mounting the single carrier in a fixed locationrelative to the substrate.

In a further embodiment again, the plurality of sensors is positionedproximate a first side of the single carrier and each of the pluralityof lens barrels is mounted to a second opposite side of the singlecarrier.

In another embodiment again, the at least one hole in the single carrierare spaced to enable an active alignment tool to grip the lens barrelduring the active alignment process.

In a further additional embodiment, mounting the single carrier relativeto at least one sensor includes mounting the single carrier relative toa single sensor.

In another additional embodiment, the single sensor is mounted to afirst side of the single carrier; and each of the plurality of lensbarrels is mounted to a second opposite side of the single carrier.

In a still yet further embodiment, the at least one hole in the singlecarrier are spaced to enable an active alignment tool to grip the lensbarrel during the active alignment process.

In still yet another embodiment, flip chip mounting is utilized to mountthe single sensor to the single carrier.

In a still further embodiment again, the single sensor is mounted to asubstrate and mounting the single carrier relative to the single sensorcomprises mounting the single carrier in a fixed location relative tothe substrate.

In still another embodiment again, the single sensor is positionedproximate a first side of the single carrier and each of the pluralityof lens barrels is mounted to a second opposite side of the singlecarrier.

In a still further additional embodiment, the at least one hole in thesingle carrier are spaced to enable an active alignment tool to grip thelens barrel during the active alignment process.

In still another additional embodiment, forming at least one hole in atleast one carrier comprises forming a ledge in at least one hole in theat least one carrier and mounting at least one spectral filter on theledge.

A yet further embodiment again also includes mounting at least onespectral filter within at least one hole in the at least one carrier.

In yet another embodiment again, the at least one spectral filter isselected from the group consisting of a color filter and an IR-cutfilter.

A further additional embodiment again also includes mounting aninterface device relative to the at least one carrier.

In another additional embodiment again, the interface device is mountedto the carrier.

In another further embodiment, the at least one sensor and the interfacedevice are mounted to a substrate and mounting the at least one carrierrelative to the at least one sensor comprises mounting the at least onecarrier in a fixed location relative to the substrate.

In still another further embodiment, independently mounting a pluralityof lens barrels to the at least one carrier comprises using activealignment to separately mount each of the lens barrels to one of the atleast one carrier.

In yet another further embodiment, the at least one hole in the at leastone carrier are spaced to enable an active alignment tool to grip thelens barrel during the active alignment process.

In another further embodiment again, the at least one opening in themodule cap are dimensioned so that the module cap is not visible withinthe field of view of any of the lens stacks and so that light does notreflect from the module cap into the lens stacks.

In another further additional embodiment, the module cap is mounted tothe at least one carrier so that a small air gap exists between themodule cap and the top of the lens barrels and the method furthercomprises applying a small bead of adhesive to each of the lens barrelsto seal the air gap between the module cap and the lens barrels.

In still another further embodiment again, the carrier is constructedfrom a material selected from the group consisting of ceramic and glass.

Still another further additional embodiment includes: forming aplurality of holes in carrier; mounting the carrier relative to aplurality of sensors so that light passing through each of the pluralityof holes in the carrier is incident on one of a plurality of focalplanes formed by the plurality of sensors; mounting at least onespectral filter within at least one of the plurality of holes in thecarrier; independently mounting a plurality of lens barrels to thecarrier, so that a lens stack in each lens barrel directs light throughthe at least one hole in the at least one carrier and focuses the lightonto a focal plane formed by a corresponding sensor in the plurality ofsensors; and mounting a module cap over the lens barrels so that themodule cap is attached to the carrier and a small air gap exists betweenthe module cap and the top of the lens barrels, where the module capincludes a plurality of openings that each admits light into one of theplurality lens stacks contained within the plurality of lens barrels;and applying a small bead of adhesive to each of the lens barrels toseal the air gap between the module cap and the lens barrels.

An array camera module in accordance with an embodiment of the inventionincludes at least one carrier in which at least one window is formed; atleast one sensor mounted relative to the at least one carrier so thatlight passing through the at least one window in the at least onecarrier is incident on a plurality of focal planes formed by at leastone array of pixels on the at least one sensor; a plurality of lensbarrels mounted to the at least one carrier, so that a lens stack ineach lens barrel directs light through the at least one window in the atleast one carrier and focuses the light onto one of the plurality offocal planes; and a module cap mounted over the lens barrels, where themodule cap includes at least one opening that admits light into the lensstacks contained within the plurality of lens barrels.

In a further embodiment, the at least one carrier is a single carrier.

In another embodiment, each of the plurality of sensors is mounted to afirst side of the single carrier; each of the plurality of lens barrelsis mounted to a second opposite side of the single carrier; and theplurality of sensors comprises a separate sensor for each of theplurality of lens barrels.

In a still further embodiment, the plurality of sensors is mounted to asubstrate and the single carrier is mounted in a fixed location relativeto the substrate; and the plurality of sensors is positioned proximate afirst side of the single carrier and each of the plurality of lensbarrels is mounted to a second opposite side of the single carrier.

In still another embodiment, the at least one sensor is a single sensor.

In a yet further embodiment, the single sensor is mounted to a firstside of the single carrier; and each of the plurality of lens barrels ismounted to a second opposite side of the single carrier.

In yet another embodiment, the single sensor is mounted to a substrateand the single carrier is mounted in a fixed location relative to thesubstrate; and the single sensor is positioned proximate a first side ofthe single carrier and each of the plurality of lens barrels is mountedto a second opposite side of the single carrier.

In a further embodiment again, the at least one sensor is mounted to asubstrate and each of a plurality of carriers is mounted in a fixedlocation relative to the substrate; and each of the plurality of lensbarrels is mounted to a separate carrier.

In another embodiment again, each lens barrel forms a separate aperture.

In a further additional embodiment, each lens barrel and correspondingfocal plane forms a camera; different cameras within the array cameramodule image different parts of the electromagnetic spectrum; and thelens stacks contained within the lens barrels differ depending upon theportion of the electromagnetic spectrum imaged by the camera to whichthe lens barrel belongs.

In another additional embodiment, the lens stacks contained within thelens barrels differ with respect to at least one of: the materials usedto construct the lens elements within the lens stacks; the shapes of atleast one surface of corresponding lens elements in the lens stacks.

In a still further embodiment again, each lens stack in the lens barrelshas a field of view that focuses light so that the plurality of arraysof pixels that form the focal planes sample the same object space withina scene.

In still another embodiment again, the pixel arrays of the focal planesdefine spatial resolutions for each pixel array; the lens stacks focuslight onto the focal planes so that the plurality of arrays of pixelsthat form the focal planes sample the same object space within a scenewith sub-pixel offsets that provide sampling diversity; and the lensstacks have modulation transfer functions that enable contrast to beresolved at a spatial frequency corresponding to a higher resolutionthan the spatial resolutions of the pixel arrays.

In a yet further embodiment again, at least one window in the at leastone carrier includes a spectral filter.

In yet another embodiment again, at least one window in at least onecarrier comprises a ledge on which the at least one spectral filter ismounted.

In a still further additional embodiment, the at least one spectralfilter is selected from the group consisting of a color filter and anIR-cut filter.

In still another additional embodiment, at least one spectral filter isapplied to an array of pixels forming a focal plane on at least one ofthe sensors.

In a yet further additional embodiment, at least one lens stack includesat least one spectral filter.

In yet another additional embodiment, the plurality of lens barrels andthe plurality of focal planes form an M×N array of cameras.

In a still further additional embodiment again, the plurality of lensbarrels and the plurality of focal planes form a 3×3 array of cameras.

In still another additional embodiment again, the M×N array of camerascomprises a 3×3 group of cameras including: a central reference camera;four cameras that capture image data in a first color channel located inthe four corners of the 3×3 group of cameras; a pair of cameras thatcapture image data in a second color channel located on either side ofthe central reference camera; and a pair of cameras that capture imagedata in a third color channel located on either side of the centralreference camera.

In another further embodiment, the reference camera is selected from thegroup consisting of: a camera including a Bayer filter; and a camerathat captures image data in the first color channel.

Still another further embodiment also includes an interface device incommunication with the at least one sensor, where the interface devicemultiplexes data received from the at least one sensor and provides aninterface via which multiplexed data is read and the imaging parametersof the focal planes formed by the at least one pixel array on the atleast one sensor are controlled.

In yet another further embodiment, the interface device is mounted tothe carrier and the carrier includes circuit traces that carry signalsbetween the interface device and the at least one sensor; and a commonclock signal coordinates the capture of image data by the at least onesensor and readout of the image data from the at least one sensor viathe interface device.

In another further embodiment again, the at least one sensor and theinterface device are mounted to a substrate, which includes circuittraces that carry signals between the interface device and the at leastone sensor; the at least one carrier is mounted in a fixed locationrelative to the at least one sensor; and a common clock signalcoordinates the capture of image data by the at least one sensor andreadout of the image data from the at least one sensor via the interfacedevice.

In another further additional embodiment, the module cap is mounted tothe at least one carrier so that a small air gap exists between themodule cap and the top of the lens barrels and a small bead of adhesiveseals the air gaps between the module cap and the lens barrels.

In still yet another further embodiment, the carrier is constructed froma material selected from the group consisting of ceramic and glass.

Still another further embodiment again includes: a carrier in which aplurality of windows are formed; a plurality of sensors each includingan array of pixels, where the plurality of sensors are mounted relativeto the carrier so that light passing through the plurality of windows isincident on a plurality of focal planes formed by the arrays of pixels;a plurality of lens barrels mounted to the at least one carrier so thata lens stack in each lens barrel directs light through the at least onewindow in the at least one carrier and focuses the light onto one of theplurality of focal planes; and a module cap mounted over the lensbarrels, where the module cap includes at least one opening that admitslight into the lens stacks contained within the plurality of lensbarrels.

An embodiment of an array camera includes: a processor; memorycontaining an image capture application; an array camera module,comprising: at least one carrier in which at least one window is formed;at least one sensor mounted relative to the at least one carrier so thatlight passing through the at least one window in the at least onecarrier is incident on a plurality of focal planes formed by at leastone array of pixels on the at least one sensor; a plurality of lensbarrels mounted to the at least one carrier, so that a lens stack ineach lens barrel directs light through the at least one window in the atleast one carrier and focuses the light onto one of the plurality offocal planes; and a module cap mounted over the lens barrels, where themodule cap includes at least one opening that admits light into the lensstacks contained within the plurality of lens barrels. In addition, theimage capture application directs the processor to: trigger the captureof image data by the array camera module; obtain and store image datacaptured by the array camera module, where the image data forms a set ofimages captured from different viewpoints; select a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints; normalize the set of images to increase the similarity ofcorresponding pixels within the set of images; determine depth estimatesfor pixel locations in an image from the reference viewpoint using atleast a subset of the set of images, wherein generating a depth estimatefor a given pixel location in the image from the reference viewpointcomprises: identifying pixels in the at least a subset of the set ofimages that correspond to the given pixel location in the image from thereference viewpoint based upon expected disparity at a plurality ofdepths; comparing the similarity of the corresponding pixels identifiedat each of the plurality of depths; and selecting the depth from theplurality of depths at which the identified corresponding pixels havethe highest degree of similarity as a depth estimate for the given pixellocation in the image from the reference viewpoint.

In a further embodiment, each lens barrel forms a separate aperture.

In another embodiment, the lens stacks contained within the lens barrelsdiffer with respect to at least one of: the materials used to constructthe lens elements within the lens stacks; the shapes of at least onesurface of corresponding lens elements in the lens stacks.

In a still further embodiment, the image capture application furtherdirects the processor to fuse pixels from the set of images using thedepth estimates to create a fused image having a resolution that isgreater than the resolutions of the images in the set of images by:determining the visibility of the pixels in the set of images from thereference viewpoint by: identifying corresponding pixels in the set ofimages using the depth estimates; and determining that a pixel in agiven image is not visible in the reference viewpoint when the pixelfails a photometric similarity criterion determined based upon acomparison of corresponding pixels; and applying scene dependentgeometric shifts to the pixels from the set of images that are visiblein an image from the reference viewpoint to shift the pixels into thereference viewpoint, where the scene dependent geometric shifts aredetermined using the current depth estimates; and fusing the shiftedpixels from the set of images to create a fused image from the referenceviewpoint having a resolution that is greater than the resolutions ofthe images in the set of images.

In a yet further embodiment, the image capture application furtherdirects the processor to synthesize an image from the referenceviewpoint by performing a super-resolution process based upon the fusedimage from the reference viewpoint, the set of images captured fromdifferent viewpoints, the current depth estimates, and visibilityinformation.

In yet another embodiment, the plurality of images comprises image datain multiple color channels; and the image capture application directsthe processor to compare the similarity of pixels that are identified ascorresponding at each of the plurality of depths by comparing thesimilarity of the pixels that are identified as corresponding in each ofa plurality of color channels at each of the plurality of depths.

In a further embodiment again, the array camera module further comprisesan interface device in communication with the at least one sensor, wherethe interface device multiplexes data received from the sensors andprovides an interface via which the processor reads multiplexed data andvia which the processor controls the imaging parameters of the focalplanes formed by the plurality of pixel arrays.

In another embodiment again, the interface device is mounted to thecarrier and the carrier includes circuit traces that carry signalsbetween the interface device and the at least one sensor; and a commonclock signal coordinates the capture of image data by the at least onesensor and readout of the image data from the at least one sensor viathe interface device.

In a further embodiment again, the at least one sensor and the interfacedevice are mounted to a substrate, which includes circuit traces thatcarry signals between the interface device and the at least one sensor;the at least one carrier is mounted in a fixed location relative to theat least one sensor; and a common clock signal coordinates the captureof image data by the at least one sensor and readout of the image datafrom the at least one sensor via the interface device.

Another further embodiment includes: a processor; memory containing animage capture application; an array camera module, comprising: a carrierin which a plurality of windows are formed; a plurality of sensors eachincluding an array of pixels, where the plurality of sensors are mountedrelative to the carrier so that light passing through the plurality ofwindows is incident on a plurality of focal planes formed by the arraysof pixels; a plurality of lens barrels mounted to the at least onecarrier so that a lens stack in each lens barrel directs light throughthe at least one window in the at least one carrier and focuses thelight onto one of the plurality of focal planes; and a module capmounted over the lens barrels, where the module cap includes at leastone opening that admits light into the lens stacks contained within theplurality of lens barrels. In addition, the image capture applicationdirects the processor to: trigger the capture of image data by the arraycamera module; obtain and store image data captured by the array cameramodule, where the image data forms a set of images captured fromdifferent viewpoints; select a reference viewpoint relative to theviewpoints of the set of images captured from different viewpoints;normalize the set of images to increase the similarity of correspondingpixels within the set of images; determine depth estimates for pixellocations in an image from the reference viewpoint using at least asubset of the set of images. Furthermore, generating a depth estimatefor a given pixel location in the image from the reference viewpointincludes: identifying pixels in the at least a subset of the set ofimages that correspond to the given pixel location in the image from thereference viewpoint based upon expected disparity at a plurality ofdepths; comparing the similarity of the corresponding pixels identifiedat each of the plurality of depths; and selecting the depth from theplurality of depths at which the identified corresponding pixels havethe highest degree of similarity as a depth estimate for the given pixellocation in the image from the reference viewpoint.

In still another further embodiment, the pixel arrays of the focalplanes define spatial resolutions for each pixel array; the lens stacksfocus light onto the focal planes so that the plurality of arrays ofpixels that form the focal planes sample the same object space within ascene with sub-pixel offsets that provide sampling diversity; and thelens stacks have modulation transfer functions that enable contrast tobe resolved at a spatial frequency corresponding to a higher resolutionthan the spatial resolutions of the pixel arrays; and the image captureapplication further directs the processor to fuse pixels from the set ofimages using the depth estimates to create a fused image having aresolution that is greater than the resolutions of the images in the setof images by: determining the visibility of the pixels in the set ofimages from the reference viewpoint by: identifying corresponding pixelsin the set of images using the depth estimates; and determining that apixel in a given image is not visible in the reference viewpoint whenthe pixel fails a photometric similarity criterion determined based upona comparison of corresponding pixels; and applying scene dependentgeometric shifts to the pixels from the set of images that are visiblein an image from the reference viewpoint to shift the pixels into thereference viewpoint, where the scene dependent geometric shifts aredetermined using the current depth estimates; and fusing the shiftedpixels from the set of images to create a fused image from the referenceviewpoint having a resolution that is greater than the resolutions ofthe images in the set of images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 conceptually illustrates an array camera in accordance with anembodiment of the invention.

FIGS. 2A-2C schematically illustrate an array camera module inaccordance with an embodiment of the invention.

FIG. 3 is a flow chart illustrating a process for manufacturing an arraycamera module in accordance with an embodiment of the invention.

FIGS. 4 and 5 conceptually illustrate the mounting of filters to acarrier during the construction of an array camera module in accordancewith an embodiment of the invention.

FIG. 6 conceptually illustrates cameras forming a π filter group inwhich red and blue cameras are located on either side of a central greencamera that can serve as a reference camera.

FIGS. 7 and 8 conceptually illustrate the mounting of sensors that eachcontain a single focal plane on a carrier during the construction of anarray camera module in accordance with an embodiment of the invention.

FIGS. 9 and 10 conceptually illustrate the mounting of lens barrels to acarrier during the construction of an array camera module in accordancewith an embodiment of the invention.

FIG. 11 conceptually illustrates an active alignment tool gripping alens barrel at a 45 degree angle relative to a 2×2 array formed by thegripped lens barrel and three adjacent lens barrels during theconstruction of an array camera module in accordance with an embodimentof the invention.

FIG. 12 conceptually illustrates attachment of a module cap to a carrierduring the construction of an array camera module in accordance with anembodiment of the invention.

FIG. 13 conceptually illustrates an array camera module including acarrier on which a 3×3 array of sensors and an interface device aremounted in accordance with an embodiment of the invention.

FIG. 14 conceptually illustrates a substrate assembly that can beutilized in the construction of an array camera module in accordancewith an embodiment of the invention.

FIG. 15 conceptually illustrates an array camera module in which asingle sensor is attached to a glass carrier in accordance with anembodiment of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, array camera modules incorporatingindependently aligned lens stacks and methods for constructing arraycamera modules incorporating independently aligned lens stacks aredescribed. An array camera is an image capture device that includesmultiple lens stacks or optical channels that direct light onto acorresponding number of focal planes, enabling the capture of multipleimages of a scene using the focal planes. The light received via each ofthe lens stacks passes through a separate aperture and so each of thecaptured images constitutes a different view of the scene. In a numberof embodiments, super-resolution processes such as those described inU.S. Patent Publication No. 2012/0147205 entitled “Systems and Methodsfor Synthesizing High Resolution Images Using Super-ResolutionProcesses”, to Lelescu et al., are utilized to synthesize a higherresolution two dimensional (2D) image or a stereo pair of higherresolution 2D images from the lower resolution images in the light fieldcaptured by an array camera. The terms high or higher resolution and lowor lower resolution are used here in a relative sense and not toindicate the specific resolutions of the images captured by the arraycamera. The disclosure within U.S. Patent Publication No. 2012/0147205concerning processes for fusing higher resolution images from a set ofimages captured from different viewpoints, synthesizing higherresolution images from a set of images captured from differentviewpoints using super-resolution processing, synthesizing highresolution images from virtual viewpoints, and for dynamicallyrefocusing synthesized high resolution images is hereby incorporated byreference in its entirety.

The term focal plane can be used to describe a region on a sensorcontaining an array of pixel elements configured to capture an imagebased upon light directed onto the focal plane via a lens stack oroptical channel. In many embodiments, each focal plane is implementedusing a separate sensor. In a number of embodiments, array cameras areimplemented using sensors that include multiple focal planes, where eachfocal plane receives light from a separate optical channel. As such, thesensor is configured to separately and (in many instances) independentlycapture and output image data from each of the focal planes.

The array cameras disclosed in U.S. Patent Publication No. 2011/0069189entitled “Capturing and Processing of Images Using Monolithic CameraArray with Heterogeneous Imagers”, to Venkataraman et al. includeexamples of array cameras in which the lens stacks of the array cameraare implemented as a single lens stack array that is aligned and mountedto a sensor. However, the large number of tolerances involved in themanufacture of a lens stack array can result in the different opticalchannels having varying focal lengths. The combination of all themanufacturing process variations typically results in a deviation of theactual (“first order”) lens parameters—such as focal length—from thenominal prescription. As a result, each optical channel can have adifferent axial optimum image location. Consequently, the lens stackarray typically cannot be placed a distance that corresponds with thefocal length of each lens stack within an array camera module. Notably,these manufacturing tolerances may result in different focal lengthseven as between lens stack arrays fabricated from the same manufacturingprocess. The disclosure within U.S. Patent Publication No. 2011/0069189regarding the implementation of different array camera architecturesincluding monolithic array cameras, non-monolithic array cameras, andarrays of array cameras is hereby incorporated by reference herein inits entirety. Array cameras in accordance with embodiments of theinvention are constructed by independently aligning each lens stack withrespect to a corresponding focal plane. In this way, each lens stack canbe optimally aligned with respect to a corresponding focal plane.

In several embodiments, an active alignment process is utilized to aligneach lens stack with respect to its corresponding focal plane. In thecontext of the manufacture of camera systems, the term active alignmenttypically refers to a process for aligning an optical system (e.g. alens stack array) with an imaging system (e.g. comprising a monolithicsensor) to achieve a final desirable spatial arrangement by evaluatingthe efficacy of the configuration as a function of the spatialrelationship between the optical system and the imaging system.Typically, this process is implemented by using the configuration tocapture and record image data (typically of a known target) in real timeas the optical system is moving relative to the imaging system. As theoptical system is moved relative to the imaging system, the spatialrelationship between the two changes, and the characteristics of therecorded image data also change correspondingly. This recorded imagedata may then be used to align the optical system relative to theimaging system in a desired manner. For example, active alignment cangenerally be used to determine a spatial relationship that results in acamera that is capable of recording images that exceed a threshold imagequality.

In several embodiments, an array camera module is constructed using aceramic carrier in which windows through the ceramic carrier are formed.A single sensor or multiple sensors can be fixed to one side of theceramic carrier to form the focal planes of the array camera module andlens barrels containing lens stacks can be affixed to the other side ofthe ceramic carrier so that the lens stacks direct light onto the focalplanes of the one or more sensors through the openings in the ceramiccarrier. The ceramic carrier is rigid and can have a co-efficient ofthermal expansion (CTE) selected to match the CTE of the sensor. In thisway, the ceramic carrier reduces the likelihood that mismatches inthermal expansion will result in changes in the alignment between thelens stacks and corresponding focal planes that deteriorate the qualityof the images that can be synthesized using the image data captured bythe focal planes. In other embodiments, any of a variety of substratematerials exhibiting rigidity and low CTE can be utilized as asubstitute for a ceramic carrier including (but not limited to) atransparent glass substrate. Furthermore, a variety of mountingtechniques can be utilized including (but not limited to) mounting oneor more sensors to a substrate and mounting the lens barrels containingthe lens stacks to a carrier, or mounting individual camera modules to asubstrate. Array cameras constructed using array camera modulesincorporating independently aligned lens stacks and methods forconstructing array camera modules incorporating independently alignedlens stacks are discussed further below.

Array Cameras Including Modules Incorporating Independently Aligned LensStacks

An array camera including an array camera module in accordance with anembodiment of the invention is illustrated in FIG. 1. The array camera100 includes an array camera module 102 including an array of cameras104. The array camera module 102 is configured to communicate with aprocessor 108, which can execute an image capture application stored asnon-transitory machine readable instructions within a memory 110. Thememory 110 can also be utilized to store image data captured by thearray camera module.

The array camera module 102 includes an array of focal planes on whichimages are formed by an array of lens stacks. Each lens stack creates anoptical channel that forms an image of the scene on an array of lightsensitive pixels within a corresponding focal plane. Each lens stack isindependently mounted within array camera module 102 to form a singlecamera 104 with the corresponding focal plane on which the lens stackforms an image. In many embodiments, each lens stack can be activelyaligned with respect to its corresponding focal plane to improve thequality of the image data capture by the focal plane.

The pixels within a focal plane of a camera 104 generate image data thatcan be sent from the array of cameras 104 to the processor 108. In manyembodiments, the lens stack within each optical channel have fields ofview that focus light so that pixels of each focal plane sample the sameobject space or region within the scene. In several embodiments, thelens stacks are configured so that the pixels that sample the sameobject space do so with sub-pixel offsets to provide sampling diversitythat can be utilized to recover increased resolution through the use ofsuper-resolution processes. The term sampling diversity refers to thefact that the images from different viewpoints sample the same object inthe scene but with slight sub-pixel offsets. By processing the imageswith sub-pixel precision, additional information encoded due to thesub-pixel offsets can be recovered when compared to simply sampling theobject space with a single image. In order to enable the recovery ofhigher resolution information, the lens stacks are designed to have aModulation Transfer Function (MTF) that enables contrast to be resolvedat a spatial frequency corresponding to the higher resolution and not atthe spatial resolution of the pixels that form a focal plane.

In the illustrated embodiment, the cameras 104 are configured in a 3×3array. In other embodiments, any of a variety of M×N camera arrayconfigurations can be utilized including linear arrays (i.e. 1×Narrays). Each camera 104 in the array camera module 102 is capable ofcapturing an image of the scene. The sensor elements utilized in thefocal planes of the cameras 104 can be individual light sensing elementssuch as, but not limited to, traditional CIS (CMOS Image Sensor) pixels,CCD (charge-coupled device) pixels, high dynamic range sensor elements,multispectral sensor elements and/or any other structure configured togenerate an electrical signal indicative of light incident on thestructure. In many embodiments, the sensor elements of each focal planehave similar physical properties and receive light via the same opticalchannel and color filter (where present). In several embodiments, thesensor elements have different characteristics and, in many instances,the characteristics of the sensor elements are related to the colorfilter applied to each sensor element.

In a variety of embodiments, color filters in individual cameras can beused to pattern the camera module with π filter groups as furtherdiscussed in U.S. Patent Publication No. 2013/0293760 entitled “CameraModules Patterned with pi Filter Groups”, the disclosure from whichrelated to filter patterns that can be utilized in the implementation ofan array camera is incorporated by reference herein in its entirety. Anyof a variety of color filter configurations can be utilized wherecameras in each color channel are distributed on either side of thecenter of the camera. The cameras can be used to capture data withrespect to different colors, or a specific portion of the spectrum. In anumber of embodiments, cameras image in the near-IR, IR, and/or far-IRspectral bands. In contrast to applying color filters to the pixels ofthe camera, color filters in many embodiments of the invention aremounted to a ceramic carrier to which one or more sensors and/or thelens stacks are mounted, or included in the lens stack. Where thesensor(s) and lens stacks are mounted to a glass substrate, the colorfilters can be applied to the glass substrate. For example, a greencolor camera can include a lens stack with a green light filter thatallows green light to pass through the optical channel. In manyembodiments, the pixels in each focal plane are the same and the lightinformation captured by the pixels is differentiated by the colorfilters in the corresponding lens stack for each filter plane. Theinclusion of spectral filters within array camera modules in accordancewith various embodiments of the invention can be implemented in avariety of other ways including (but not limited to) by applying colorfilters to the pixels of the focal planes of the cameras similar to themanner in which color filters are applied to the pixels of aconventional color camera. In several embodiments, at least one of thecameras in the camera module can include uniform color filters appliedto the pixels in its focal plane. In many embodiments, a Bayer filterpattern is applied to the pixels of at least one of the cameras in acamera module. In a number of embodiments, camera modules areconstructed in which color filters are utilized in both the lens stacksand on the pixels of the imager array.

In several embodiments, the processor 108 is configured to take theimage data captured by the sensor and synthesize high resolution images.In a number of embodiments, the processor 108 is configured to measuredistances to or depth of objects in the scene using the set of imagescaptured by the array camera module. In many embodiments, the process ofsynthesizing high resolution images from a set of images captured by thearray camera module also involves generating depth information withrespect to objects visible within the field of view of the array camera.U.S. Pat. No. 8,619,082 entitled “Systems and Methods for ParallaxDetection and Correction in Images Captured Using Array Cameras thatContain Occlusions using Subsets of Images to Perform Depth Estimation”to Ciurea et al. discloses techniques for estimating depth using sets ofimages captured from different viewpoints. The disclosure within U.S.Pat. No. 8,619,082 concerning estimating depth and generating a depthmap using multiple images of a scene and synthesizing images fromdifferent perspectives using depth information is also incorporated byreference herein in its entirety. In many embodiments of the invention,the process of estimating depth and/or synthesizing a higher resolutionimage of a scene from a set of images involves selection of a referenceviewpoint, typically that of a reference camera.

In many embodiments, a set of images is created using the image datacaptured by the cameras in the array camera module and can be consideredto be a number of images of the scene captured from differentviewpoints. In order to assist with depth estimation and/or synthesis ofhigher resolution images, the set of images can be normalized toincrease the similarity of corresponding pixels within the images. Inseveral embodiments, the process of estimating depth and/or building adepth map of the scene from the reference viewpoint involves determiningdepth estimates for pixel locations in an image from the referenceviewpoint using at least a subset of the set of images, whereingenerating a depth estimate for a given pixel location in the image fromthe reference viewpoint includes: identifying pixels in the at least asubset of the set of images that correspond to the given pixel locationin the image from the reference viewpoint based upon expected disparityat a plurality of depths; comparing the similarity of the correspondingpixels identified at each of the plurality of depths; and selecting thedepth from the plurality of depths at which the identified correspondingpixels have the highest degree of similarity as a depth estimate for thegiven pixel location in the image from the reference viewpoint. When thearray camera module captures image data in multiple color channels, thearray camera can compare the similarity of pixels that are identified ascorresponding at each of the plurality of depths by comparing thesimilarity of the pixels that are identified as corresponding in each ofthe color channels at each of the plurality of depths. These processesare discussed in more detail in U.S. Pat. No. 8,619,082, the relevantdisclosure of which is incorporated by reference herein and above byreference in its entirety.

In a number of embodiments, a higher resolution image is synthesizedfrom the set of images obtained from the array camera module by fusingpixels from the set of images using the depth estimates to create afused image having a resolution that is greater than the resolutions ofthe images in the set of images. The fusion process can include:identifying the pixels from the set of images that are visible in animage from the reference viewpoint using the at least one visibilitymap; applying scene dependent geometric shifts to the pixels from theset of images that are visible in an image from the reference viewpointto shift the pixels into the reference viewpoint, where the scenedependent geometric shifts are determined using the current depthestimates; and fusing the shifted pixels from the set of images tocreate a fused image from the reference viewpoint having a resolutionthat is greater than the resolutions of the images in the set of images.In several embodiments, the process of synthesizing a higher resolutionimage involves performing an additional super-resolution process basedupon the fused image from the reference viewpoint, the set of imagescaptured from different viewpoints, the current depth estimates, and thevisibility information. The fusion and super-resolution processes aredescribed in more detail in U.S. Patent Publication No. 2012/0147205 therelevant disclosure of which is incorporated by reference herein andabove in its entirety.

In many embodiments, the processor 108 is able to synthesize an imagefrom a virtual viewpoint. In a number of embodiments, a virtualviewpoint is any viewpoint which is not the reference viewpoint. Inseveral embodiments, the virtual viewpoint corresponds to a viewpoint ofone of the cameras 104 in the array camera module 102 that is not thereference camera. In many embodiments, the processor is able tosynthesize an image from a virtual viewpoint, which does not correspondto any camera 104 in the array camera module 102.

Although specific array camera architectures are described above withrespect to FIG. 1, alternative architectures can also be utilized inaccordance with embodiments of the invention. Array camera modulesincluding independently aligned lens stacks in accordance withembodiments of the invention and discussed further below.

Array Camera Modules

Array camera modules incorporating independently aligned lens stacks canoffer a variety of benefits including (but not limited to) capturingimage data using focal planes that are located at the back focal lengthof their corresponding lens stacks. In addition, array camera modulesconstructed in accordance with many embodiments of the inventioninterpose materials between sensors and lens barrels containing the lensstacks that reduce the impact of CTE mismatch between the low CTEsemiconductor materials from which the sensors are fabricated and thehigher CTE materials from which the lens barrels are constructed.Accordingly, array camera modules can be constructed that achieveprecise alignment of optics and robustness to variations in thermalconditions.

An array camera module incorporating independently aligned lens stacksin accordance with an embodiment of the invention is illustrated inFIGS. 2A-2C. FIGS. 2B and 2C illustrate cross sections of an arraycamera module 102 illustrated in FIG. 2A taken along an axis 164. Thearray camera module 102 includes a carrier 300, which can be implementedusing a ceramic carrier and/or any of a variety of materials possessingrigidity and low CTE that are appropriate to the requirements of aspecific application. Windows extend from a first side 302 through to asecond side 204 of the carrier 300. Windows can be holes and ortransparent regions of the carrier. In the illustrated embodiment, thewindows are rectangular holes and color filters and/or IR cut-offfilters 305 are mounted within the opening of each hole. As discussedabove, the inclusion of spectral filters in openings within the carrieris optional and spectral filters can be located within the lens barrelsand/or on the sensor elements of the focal planes. At least one sensor310 is mounted on the first side 302 of the carrier so that the sensorpixels are positioned facing inward to receive light that passes throughthe color filter 305 mounted within the carrier 300. In the illustratedembodiment, a single sensor is shown per camera. As noted above, asingle sensor can form the focal planes of multiple cameras. In manyembodiments, a single sensor forms the focal planes of all of thecameras in the array. In several embodiments, the single sensor includesa single array of pixels that is read out to capture an image from eachof the optical channels formed by the lens barrels. In many embodiments,the single sensor includes a separate independently controllable arrayof pixels that form the focal planes of each of the cameras. A lensbarrel 320 containing a lens stack is mounted on the second side 304 ofthe carrier 300. The lens barrel forms an aperture and each lens barrel320 is positioned so that the outermost lens 322 of the lens stackcontained within the lens barrel directs light into the lens stack. Inmany embodiments, cameras in the array camera module image differentparts of the electromagnetic spectrum and the lens stacks containedwithin the lens barrels differ depending upon the color channel to whichthe camera belongs. In several embodiments, the surfaces of the lenselements, and/or the material used in the construction of the lenselements within the lens stacks differ based upon the portion of thespectrum imaged by a camera. A module cap 330 is fixed to the carrier300 and extends over the lens barrels 320. In the illustratedembodiment, the outermost lens 322 contained within each lens barrel 320receives light through an opening in the module cap 330. The openings inthe module cap 330 can be dimensioned to avoid the module cap 330 fromobscuring the fields of view of the lens stacks and/or reflecting lightinto the lens stacks. In other embodiments, the module cap can includeone or more cover glasses through which the lens barrels can receivelight.

Although the array camera modules discussed above with respect to FIGS.2A-2C utilize a separate sensor mounted to the carrier for each camerain the array camera module, a single sensor or multiple sensorsincorporating more than one focal plane can be mounted to one side ofcarrier with a separate lens barrel for each camera mounted to the otherside of the carrier. The sensor(s) need not be mounted to the samecarrier as the lens barrels. The substrate to which the sensor(s) aremounted, whether the carrier or a separate substrate, can includecircuit traces that provide power to the sensor(s) and enable read outof data. Furthermore, the sensor(s) can be mounted to a substrate andindependent carriers can be utilized to mount the lens barrels to thesensor(s). In embodiments that utilize multiple sensors, the sensors cancommunicate with another device mounted to the substrate thatmultiplexes data received from the sensors and provides an interface viawhich a processor can read out multiplexed data and control the imagingparameters of the focal planes within the array camera module. Processesfor constructing array camera modules in accordance with embodiments ofthe invention are discussed further below.

Manufacturing Array Camera Modules

A variety of processes can be utilized to construct array camera modulesin accordance with embodiments of the invention and the specificprocesses that are utilized typically depend upon the materials utilizedin the construction of the array camera module and the manner in whichone or more sensors and/or each camera's lens barrel is mounted. In anumber of embodiments, the process of manufacturing an array cameramodule includes independently actively aligning each lens barrel.

A process for manufacturing an array camera module utilizing a carrierto which one or more sensors and camera lens barrels are independentlymounted using active alignment in accordance with an embodiment of theinvention is illustrated in FIGS. 3-13. The process 350 includesmanufacturing (360) a carrier. As noted above with respect to FIGS.2A-2C, in many embodiments each camera is formed around a window in thecarrier that enables a lens stack contained within a lens barrel todirect light onto the focal plane of a sensor. Furthermore, a colorfilter and/or an IR cut-off filter can be mounted within an opening inthe carrier that forms window. Accordingly, the process of manufacturingthe carrier includes forming the appropriate windows, which can involveforming holes through the carrier or applying light blocking masks to atransparent carrier to define transparent windows through the carrier.In several embodiments, ledges are formed within the holes to facilitatethe mounting of color filters and/or IR cut-off filters within the hole.

Referring again to FIG. 3 and FIGS. 4 and 5, color filters and/oradditional filters such as (but not limited to) IR cut-off filters canbe mounted (362) so that light passes through the filters in order topass from one side of the carrier through a window to the other side ofthe carrier. In the embodiment illustrated in FIGS. 4 and 5, green 306,blue 307, and red 308 color filters are inserted along with IR cut-offfilters into holes 301 formed within the carrier 300. The carrier isconfigured to be incorporated into an array camera module containingnine cameras and green color filters are incorporated within five of thecameras, two cameras incorporate blue color filters 307 and two camerasincorporate red color filters 308. The configuration of the camerasforms the π filter group conceptually illustrated in FIG. 6 in which redand blue cameras are located equidistant and on either side of a centralgreen camera that can serve as a reference camera. In other embodiments,any of a variety of filters applied in any of a variety of patterns canbe utilized as appropriate to the requirements of specific applications.Furthermore, many applications do not involve the application of filtersto the carrier. For example, an array camera module that includes one ormore Bayer cameras can utilize one or more sensors to which colorfilters are directly applied. In many embodiments, the techniquesdescribed above are utilized to form a π filter group with a centralBayer camera or an array in which each camera is a Bayer camera. Thespecific selection of filters typically depends upon the requirements ofa specific application.

Referring again to FIG. 3, one or more sensors are mounted (364) to thecarrier so that light passing through the windows in the carrier isincident on the focal planes of the sensor(s). The mounting of sensorsthat each contain a single focal plane is illustrated in FIGS. 7 and 8.Each sensor 310 is mounted to a first side 302 of the carrier 300. Inmany embodiments, the color filters are mounted within or covering theopening on the second side 304 of the carrier. In many embodiments, flipchip mounting is utilized to mount the one or more sensors to thecarrier. As can readily be appreciated, the mounting of one or moresensors to the carrier is optional. Sensors can be mounted to anothersubstrate that is fixed in a location relative to a carrier to which thelens barrels of the cameras are mounted at some stage during theconstruction of the array camera module.

Referring again to FIG. 3, the process 350 includes independentlymounting (366) each of the lens barrels of the cameras to the carrier.The mounting of lens barrels 320 to the second surface 304 of thecarrier 300 is conceptually illustrated in FIGS. 9 and 10. In manyembodiments, active alignment is utilized to align each of the lensbarrels to the carrier. In order to facilitate active alignment, thewindows in the carrier which define the mounting locations of the lensbarrels and the dimensions of the lens barrels are determined to enablean active alignment tool to grip the lens barrel during activealignment.

In several embodiments, lens barrels can be placed close together byutilizing the ability of an active alignment tool to grip a lens barrellocated adjacent three other lens barrels in a 2×2 array at an anglerelative to the rows and columns of the 2×2 array. In this way, thegripper of the active alignment tool does not need to extend through thenarrowest portion of the gap between any two of the adjacent lensbarrels in order to place a lens barrel. Therefore, the gap between anytwo adjacent lens barrels is not dependent upon the dimensions of thegripper of the active alignment tool. An active alignment tool grippinga lens barrel at a 45 degree angle relative to a 2×2 array formed by thegripped lens barrel and three adjacent lens barrels in accordance withan embodiment of the invention is illustrated in FIG. 11. As can readilybe appreciated the width of each member 370 of the gripper is greaterthan the spacing between adjacent lens barrels. By clasping the membersof the gripper so that they contact lens barrel 9 at an angle relativeto the rows and columns of the array of lens barrels, the members 370 ofthe gripper need not extend into the narrowest portion of the gapbetween adjacent lens barrels (2) and (5). While the axis on which thelens barrel is gripped in the illustrated embodiment is at a 45 degreeangle relative to the axes of the rows and columns of the lens barrelarray, the specific angle of the axis on which the lens barrel isgripped relative to the axes of the rows and columns of the lens barrelarray is largely determined based upon the dimensions of the gripper ofthe active alignment machine and the available spacing between adjacentlens barrels. In the illustrated embodiment, the lens barrels 320 arenumbered to indicate the order in which the lens barrels were placed onthe carrier 300. The first lens barrel (1) placed on the carrier usingthe active alignment machine was placed in the center of the carrier.Lens barrels (2), (3), (4), and (5) were then placed in the remainingpositions within a 3×3 array that are not corners. Finally, lens barrels(6), (7), (8), and (9) were placed in the corners of the 3×3 array.

Although a specific sequence is illustrated in FIG. 11, alternativesequences can be utilized in which a pair of lens barrels is identified(4) and (7) and a third lens barrel (10) is placed using activealignment to form an L shape. A 2×2 array of lens barrels (4), (7),(10), and (11) can then be formed by positioning the gripper to contactthe lens barrel of a fourth lens barrel (11) along an axis at an anglerelative to the axis of the rows and columns of the 2×2 array. Theprocess can then be repeated with respect to each adjacent pair of lensbarrels (e.g. lens barrels 7 and 3) and/or each L shaped group of lensbarrels formed by the placement of lens barrels (e.g. lens barrels 4, 6,and 10). Provided that the active alignment tool is not required toattempt to place a lens barrel between two lens barrels to form threelens barrels aligned along an axis, the spacing between the lens barrelscan be determined in a manner that is not related to the dimensions ofthe gripper used to clasp the lens barrel during the active alignmentprocess.

While specific processes for independently aligning lens stacks withinan array camera module are described above with respect to FIGS. 9-11,any of a variety of techniques including passive and/or active alignmentprocesses can be utilized as appropriate to the requirements of specificapplications in accordance with embodiments of the invention.

Referring back to FIG. 3, a module cap can be mounted (368) over thelens barrels and attached to the carrier to protect the lens barrels.Attachment of a module cap to a carrier in accordance with an embodimentof the invention is illustrated in FIG. 12. As noted above, the modulecap 330 includes openings 332 that admit light into the lens stackscontained within the lens barrels 320. Ideally, the openings in themodule cap are dimensioned so that the module cap is not visible withinthe field of view of any of the lens stacks and/or so that light doesnot reflect from the module cap into the lens stacks. In severalembodiments, a small air gap exists between the module cap and the topof the lens barrels. A small bead of adhesive can be applied to each ofthe lens barrels to seal the air gap between the module cap and the lensbarrels. In a number of embodiments, the module cap is constructed froma low CTE polymer such as (but not limited to) a glass filled liquidcrystal polymer. By utilizing a low CTE polymer, warping of the lensbarrels due to CTE mismatch between the carrier and the module cap canbe avoided. In other embodiments, the module cap can be constructed fromany material appropriate to the requirements of a specific application.

Although a variety of processes are described above in reference toFIGS. 3-12 with respect to the manufacture of array camera modules,various aspects of the processes described above can be utilized toconstruct any of a variety of array camera modules that incorporate anarray of independently aligned lens stacks as appropriate to therequirements of specific applications in accordance with embodiments ofthe invention. Furthermore, not all of the components discussed aboveneed to be mounted to the same carrier and/or additional components canbe mounted to carriers in accordance with embodiments of the invention.Array camera modules that incorporate an interface device mounted withinthe array camera module to enable communication with a processor inaccordance with embodiments of the invention are discussed furtherbelow.

Interfacing with External Devices

Reading image data from an array camera module can involve reading imagedata from each of the active sensors within the array camera module. Theprocess of communicating with each of the sensors in an array cameramodule can be simplified by utilizing a separate interface device thatis responsible for multiplexing image data received from multiplesensors for output to an external device and for controlling imagingparameters of individual sensors in response to commands received fromexternal devices. In a number of embodiments, the substrate or carrierto which the sensors are mounted includes electrical traces that can beutilized to carry signals between the sensors and the interface device.

An array camera module including a carrier on which a 3×3 array ofsensors and an interface device are mounted is illustrated in FIG. 13.The 3×3 array of sensors 310 and the interface device 400 are mounted toa carrier 300 on which circuit traces 402 are patterned. In a number ofembodiments, the sensors 310 communicate with the interface device 400using low-voltage differential signaling (LVDS). In several embodiments,a common clock signal coordinates the capture and readout of image databy the sensors and the interface device 400 multiplexes the capturedimage data received via the LVDS connections for output via an interfaceappropriate to a specific processor. In certain embodiments, theinterface device 400 communicates with external devices such asprocessors and/or graphics processors using a MIPI CSI 2 outputinterface supporting four lane video read-out of video at 30 fps fromthe array camera module. The bandwidth of each lane can be optimized forthe total number of pixels in the sensor(s) within the array cameramodule and the desired frame rate. The use of various interfacesincluding the MIPI CSI 2 interface to transmit image data captured by anarray of focal planes to an external device in accordance withembodiments of the invention is described in U.S. Pat. No. 8,305,456 toMcMahon, the disclosure of which is incorporated by reference herein inits entirety. In other embodiments, any interface appropriate to therequirements of specific applications can be utilized includinginterfaces that enable the control of the imaging parameters of groupsof focal planes by an external device in a manner similar to thatdescribed in U.S. Provisional Patent Publication No. 2014/0132810entitled “Systems and Methods for Array Camera Focal Plane Control” toMcMahon, filed Nov. 13, 2013, the disclosure of which is incorporated byreference herein in its entirety.

In embodiments where one or more sensors are mounted to a separatesubstrate to the carrier, an interface device can also be mounted to thesubstrate. A substrate assembly that can be utilized in the constructionof an array camera module in accordance with an embodiment of theinvention is illustrated in FIG. 14. The substrate assembly comprises asubstrate 410 to which multiple sensors 310 and an interface device 400are attached. The substrate 410 is bonded to a carrier 300 to which lensbarrels can be independently mounted utilizing processes similar tothose outlined above. In many instances the substrate is bonded to thecarrier and the windows through the carrier are dimensioned to providesufficient tolerances to ensure that the focal planes of each of thesensors are positioned within the openings. Although the illustratedembodiment includes multiple sensors, a similar configuration can alsobe utilized with a single sensor that forms multiple focal planes (theimaging parameters and read-out of which may or may not be independentlycontrolled).

In many embodiments, a single sensor is utilized. A camera module inwhich lens barrels and a sensor are mounted to a carrier in accordancewith an embodiment of the invention is illustrated in FIG. 15. Thecamera module 1500 includes four lens barrels 1502 and a single sensor1504 that are attached to a carrier 1506, which is constructed from atransparent glass material. As can readily be appreciated, a cameramodule to which lens barrels and a single sensor are attached canutilize any of a variety of carrier materials as appropriate to therequirements of specific applications in accordance with embodiments ofthe invention.

Although the present invention has been described in certain specificaspects, many additional modifications and variations would be apparentto those skilled in the art. It is therefore to be understood that thepresent invention can be practiced otherwise than specificallydescribed. Thus, embodiments of the present invention should beconsidered in all respects as illustrative and not restrictive.

What is claimed is:
 1. An array camera, comprising: a processor; memorycontaining an image capture application; an array camera module,comprising: at least one carrier in which at least one window is formed;at least one sensor mounted relative to the at least one carrier so thatlight passing through the at least one window in the at least onecarrier is incident on a plurality of focal planes formed by at leastone array of pixels on the at least one sensor; a plurality of lensbarrels mounted to the at least one carrier, so that a lens stack ineach lens barrel directs light through the at least one window in the atleast one carrier and focuses the light onto one of the plurality offocal planes; and a module cap mounted over the lens barrels, where themodule cap includes at least one opening that admits light into the lensstacks contained within the plurality of lens barrels; wherein the imagecapture application directs the processor to: trigger the capture ofimage data by the array camera module; obtain and store image datacaptured by the array camera module, where the image data forms a set ofimages captured from different viewpoints; select a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints; normalize the set of images to increase the similarity ofcorresponding pixels within the set of images; determine depth estimatesfor pixel locations in an image from the reference viewpoint using atleast a subset of the set of images, wherein generating a depth estimatefor a given pixel location in the image from the reference viewpointcomprises: identifying pixels in the at least a subset of the set ofimages that correspond to the given pixel location in the image from thereference viewpoint based upon expected disparity at a plurality ofdepths; comparing the similarity of the corresponding pixels identifiedat each of the plurality of depths; and selecting the depth from theplurality of depths at which the identified corresponding pixels havethe highest degree of similarity as a depth estimate for the given pixellocation in the image from the reference viewpoint.
 2. The array cameraof claim 1, wherein the at least one carrier is a single carrier.
 3. Thearray camera of claim 2, wherein: each of the plurality of sensors ismounted to a first side of the single carrier; each of the plurality oflens barrels is mounted to a second opposite side of the single carrier;and the plurality of sensors comprises a separate sensor for each of theplurality of lens barrels.
 4. The array camera of claim 2, wherein: theplurality of sensors is mounted to a substrate and the single carrier ismounted in a fixed location relative to the substrate; and the pluralityof sensors is positioned proximate a first side of the single carrierand each of the plurality of lens barrels is mounted to a secondopposite side of the single carrier.
 5. The array camera of claim 2,wherein the at least one sensor is a single sensor.
 6. The array cameraof claim 5, wherein: the single sensor is mounted to a first side of thesingle carrier; and each of the plurality of lens barrels is mounted toa second opposite side of the single carrier.
 7. The array camera ofclaim 5, wherein: the single sensor is mounted to a substrate and thesingle carrier is mounted in a fixed location relative to the substrate;and the single sensor is positioned proximate a first side of the singlecarrier and each of the plurality of lens barrels is mounted to a secondopposite side of the single carrier.
 8. The array camera of claim 1,wherein: the at least one sensor is mounted to a substrate and each of aplurality of carriers is mounted in a fixed location relative to thesubstrate; and each of the plurality of lens barrels is mounted to aseparate carrier.
 9. The array camera of claim 1, wherein: each lensbarrel and corresponding focal plane forms a camera; different cameraswithin the array camera module image different parts of theelectromagnetic spectrum; and the lens stacks contained within the lensbarrels differ depending upon the portion of the electromagneticspectrum imaged by the camera to which the lens barrel belongs.
 10. Thearray camera of claim 1, wherein each lens stack in the lens barrels hasa field of view that focuses light so that the plurality of arrays ofpixels that form the focal planes sample the same object space within ascene.
 11. The array camera of claim 10, wherein: the pixel arrays ofthe focal planes define spatial resolutions for each pixel array; thelens stacks focus light onto the focal planes so that the plurality ofarrays of pixels that form the focal planes sample the same object spacewithin a scene with sub-pixel offsets that provide sampling diversity;and the lens stacks have modulation transfer functions that enablecontrast to be resolved at a spatial frequency corresponding to a higherresolution than the spatial resolutions of the pixel arrays.
 12. Thearray camera of claim 11, wherein the image capture application furtherdirects the processor to fuse pixels from the set of images using thedepth estimates to create a fused image having a resolution that isgreater than the resolutions of the images in the set of images by:determining the visibility of the pixels in the set of images from thereference viewpoint by: identifying corresponding pixels in the set ofimages using the depth estimates; and determining that a pixel in agiven image is not visible in the reference viewpoint when the pixelfails a photometric similarity criterion determined based upon acomparison of corresponding pixels; and applying scene dependentgeometric shifts to the pixels from the set of images that are visiblein an image from the reference viewpoint to shift the pixels into thereference viewpoint, where the scene dependent geometric shifts aredetermined using the current depth estimates; and fusing the shiftedpixels from the set of images to create a fused image from the referenceviewpoint having a resolution that is greater than the resolutions ofthe images in the set of images.
 13. The array camera of claim 12,wherein the image capture application further directs the processor tosynthesize an image from the reference viewpoint by performing asuper-resolution process based upon the fused image from the referenceviewpoint, the set of images captured from different viewpoints, thecurrent depth estimates, and visibility information.
 14. The arraycamera of claim 1, wherein at least one spectral filter is mountedwithin at least one window in the at least one carrier.
 15. The arraycamera of claim 14, wherein the at least one spectral filter is selectedfrom the group consisting of a color filter and an IR-cut filter. 16.The array camera of claim 1, wherein at least one spectral filter isapplied to an array of pixels forming a focal plane on at least one ofthe sensors.
 17. The array camera of claim 1, wherein at least one lensstack includes at least one spectral filter.
 18. The array camera ofclaim 1, wherein: the plurality of images comprises image data inmultiple color channels; and the image capture application directs theprocessor to compare the similarity of pixels that are identified ascorresponding at each of the plurality of depths by comparing thesimilarity of the pixels that are identified as corresponding in each ofa plurality of color channels at each of the plurality of depths. 19.The array camera of claim 1, wherein the plurality of lens barrels andthe plurality of focal planes form an M×N array of cameras.
 20. Thearray camera of claim 19, wherein the plurality of lens barrels and theplurality of focal planes form a 3×3 array of cameras.
 21. The arraycamera of claim 19, wherein the M×N array of cameras comprises a 3×3group of cameras comprising: a central reference camera; four camerasthat capture image data in a first color channel located in the fourcorners of the 3×3 group of cameras; a pair of cameras that captureimage data in a second color channel located on either side of thecentral reference camera; and a pair of cameras that capture image datain a third color channel located on either side of the central referencecamera.
 22. The array camera of claim 21, wherein the reference camerais selected from the group consisting of: a camera including a Bayerfilter; and a camera that captures image data in the first colorchannel.
 23. The array camera of claim 1, wherein the array cameramodule further comprises an interface device in communication with theat least one sensor, where the interface device multiplexes datareceived from the sensors and provides an interface via which theprocessor reads multiplexed data and via which the processor controlsthe imaging parameters of the focal planes formed by the plurality ofpixel arrays.
 24. The array camera of claim 23, wherein: the interfacedevice is mounted to the carrier and the carrier includes circuit tracesthat carry signals between the interface device and the at least onesensor; and a common clock signal coordinates the capture of image databy the at least one sensor and readout of the image data from the atleast one sensor via the interface device.
 25. The array camera of claim23, wherein: the at least one sensor and the interface device aremounted to a substrate, which includes circuit traces that carry signalsbetween the interface device and the at least one sensor; the at leastone carrier is mounted in a fixed location relative to the at least onesensor; and a common clock signal coordinates the capture of image databy the at least one sensor and readout of the image data from the atleast one sensor via the interface device.
 26. The array camera of claim1, wherein the module cap is mounted to the at least one carrier so thata small air gap exists between the module cap and the top of the lensbarrels and a small bead of adhesive seals the air gaps between themodule cap and the lens barrels.
 27. The array camera of claim 1,wherein the carrier is constructed from a material selected from thegroup consisting of ceramic and glass.
 28. A array camera, comprising: aprocessor; memory containing an image capture application; an arraycamera module, comprising: a carrier in which a plurality of windows areformed; a plurality of sensors each including an array of pixels, wherethe plurality of sensors are mounted relative to the carrier so thatlight passing through the plurality of windows is incident on aplurality of focal planes formed by the arrays of pixels; a plurality oflens barrels mounted to the at least one carrier so that a lens stack ineach lens barrel directs light through the at least one window in the atleast one carrier and focuses the light onto one of the plurality offocal planes; and a module cap mounted over the lens barrels, where themodule cap includes at least one opening that admits light into the lensstacks contained within the plurality of lens barrels; wherein the imagecapture application directs the processor to: trigger the capture ofimage data by the array camera module; obtain and store image datacaptured by the array camera module, where the image data forms a set ofimages captured from different viewpoints; select a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints; normalize the set of images to increase the similarity ofcorresponding pixels within the set of images; determine depth estimatesfor pixel locations in an image from the reference viewpoint using atleast a subset of the set of images, wherein generating a depth estimatefor a given pixel location in the image from the reference viewpointcomprises: identifying pixels in the at least a subset of the set ofimages that correspond to the given pixel location in the image from thereference viewpoint based upon expected disparity at a plurality ofdepths; comparing the similarity of the corresponding pixels identifiedat each of the plurality of depths; and selecting the depth from theplurality of depths at which the identified corresponding pixels havethe highest degree of similarity as a depth estimate for the given pixellocation in the image from the reference viewpoint.
 29. The array cameraof claim 28, wherein: the pixel arrays of the focal planes definespatial resolutions for each pixel array; the lens stacks focus lightonto the focal planes so that the plurality of arrays of pixels thatform the focal planes sample the same object space within a scene withsub-pixel offsets that provide sampling diversity; and the lens stackshave modulation transfer functions that enable contrast to be resolvedat a spatial frequency corresponding to a higher resolution than thespatial resolutions of the pixel arrays; and the image captureapplication further directs the processor to fuse pixels from the set ofimages using the depth estimates to create a fused image having aresolution that is greater than the resolutions of the images in the setof images by: determining the visibility of the pixels in the set ofimages from the reference viewpoint by: identifying corresponding pixelsin the set of images using the depth estimates; and determining that apixel in a given image is not visible in the reference viewpoint whenthe pixel fails a photometric similarity criterion determined based upona comparison of corresponding pixels; and applying scene dependentgeometric shifts to the pixels from the set of images that are visiblein an image from the reference viewpoint to shift the pixels into thereference viewpoint, where the scene dependent geometric shifts aredetermined using the current depth estimates; and fusing the shiftedpixels from the set of images to create a fused image from the referenceviewpoint having a resolution that is greater than the resolutions ofthe images in the set of images.